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1.  Introduction 


Rapid  and  robust  scene  understanding  is  a  eritically  important  goal  for  the 
development  of  Army  autonomous  intelligent  systems.'  For  outdoor  natural  seenes, 
it  will  be  important  for  autonomous  intelligent  systems  to  be  able  to  quickly  discern 
the  depth  of  view,  navigability,  exposure  or  eoneealment  (as  it  relates  to  objeet 
searehing),  and  transienee,  that  is,  the  rate  at  whieh  elements  of  the  seene  or  its 
environment  are  ehanging  in  spaee  and  time.^’^  In  this  regard,  salieney  estimation 
has  been  helpful  to  eomputationally  identify  elements  in  a  seene  that  immediately 
eapture  the  visual  attention  of  an  observer."'’^  Several  reeent  papers  have  discussed 
eoneepts  associated  with  visual  salieney  to  enhance  automated  navigation  and 
seene  exploration.^"^  Note,  however,  that  the  most  aetive  or  salient  objeet(s)  in  a 
scene  may  not  represent  the  most  important  or  meaningful  feature(s)  of  the  scene. ^ 
For  example,  an  automated  vision  system  may  readily  deteet  changes  in  the  ground 
surfaee  as  a  new  or  different  object  in  the  field  of  view;  however,  reeognizing  the 
physieal  eharacteristies  of  the  new  surfaee"’’"  (e.g.,  shallow  or  deep  water,  thiek  or 
thin  iee,  snow,  mud,  quicksand,  etc.)  and  observing  any  changes  in  the 
environmental  eontext  of  the  image'^"'"'  maybe  eritieally  important.  Characterizing 
interaetions  between  objeets  and  the  environment  also  can  contribute  to  physieal 
scene  understanding.'^’'^  In  the  example  above,  if  a  scene  depiets  vehicles  or 
personnel  activity  in  a  changing  complex  environment,  then  robust  seene 
understanding  could  provide  key  border  and  aeeessibility  information  for 
navigation  and  trafficability. 

Nevertheless,  many  eurrent  methods  for  seene  understanding,  like  those  that 
generate  image  deseriptions  via  automated  semantie  labeling'^  or  visual  seene 
elassification,'*  do  not  address  image  information  (and  image  context)  affeeted  by 
changing  environmental  conditions.  Yet,  the  interpretation  of  ehanging 
environmental  eonditions  can  pose  serious  challenges  for  eomputer  vision 
proeesses,  sueh  as  those  assoeiated  with  place  reeognition,  navigation,  road/terrain 
deteetion,  and  seene  exploration. This  is  beeause  rain,  snow,  and  fog  weather 
events,  smoke,  haze,  or  other  ehanges  in  lighting  and  visibility  ean  significantly 
obscure  features,  degrade  objeet  reeognition,  and  modify  the  salieney  and  image 
eontext  of  an  outdoor  seene. Naturally,  seene-depieted  environmental 
conditions  can  vary  with  time  of  day,  season,  and  loeation.^^ 

Similar  challenges  ean  also  extend  to  interpreting  space-  and  time-changing  scenes 
due  to  visual  motion  of  objects  within  the  field  of  view.^"'"^®  In  this  ease,  ehanging 
environmental  conditions  such  as  illumination,  precipitation,  and  vegetation  ean 
make  feature  reeognition  of  moving  objeets  unclear,  so  that  identifying  moving 
objeets  in  outdoor  environments  beeomes  more  diffieult  for  vision-based  intelligent 
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systems. As  an  example,  poor  contrast  in  images  can  be  brought  about  by  low 
visibility  due  to  environmental  effects  or  weak  illumination,  such  as  during  dawn, 
dusk,  or  night.^"^  Visually  degraded  or  blurred  images  can  be  brought  about  by  rapid 
movements  of  the  camera  and/or  objects  in  the  field  of  view,  especially  in  the  low- 
light  case,  which  necessitate  longer  camera  exposure  times. In  addition,  space 
and  time  variations  in  scene  illumination  can  affect  the  optical  flow  field  in  images 
and  movies. Note  that  there  have  been  many  optical  flow  approaches  used  to 
detect  the  motion  of  objects  in  a  scene,  which  have  been  helpful  in  a  variety  of 
applications. Nevertheless,  camera  motion  may  introduce  some  unmanageable 
artifacts  with  some  of  these  gradient-based  optical  flow  approaches  if  they  are  not 
augmented  by  more  sophisticated  spatiotemporal  analyses. 

In  this  report,  we  propose  that  it  is  important  to  incorporate  space-  and  time- varying 
environmental  image  information  from  the  very  beginning  of  the  data  collection 
process  so  that  the  recorded  images  can  be  more  effectively  indexed  and  retrieved 
for  operational  use  and  analysis.  This  top-down  approach  not  only  provides  a 
systematic  characterization  of  the  measured  data  for  better  scene  description,  but 
can  help  the  end  user  (Soldier)  develop  improved  course  of  action  strategies  based 
on  scene  understanding  (algorithms  and  analysis)  incorporating  battlefield 
environments  changing  in  space  and  time.  Incorporating  space-  and  time-varying 
environmental  image  information  for  better  scene  understanding  can  be  vital  to 
support  numerous  Army  missions,"^^"^^  such  as  those  related  to  weather  elements  on 
the  battlefield  that  can  alter  terrain  features  and  trafficability;  low  visibility  that  can 
impede  reconnaissance  and  target  acquisition  or  alternately  conceal  friendly  forces 
maneuvers  and  activities;  and  wind  speed  and  direction  that  can  favor  upwind 
forces  in  nuclear,  biological,  and  chemical  (NBC)  attacks  or  decrease  the 
effectiveness  of  downwind  forces  due  blowing  dust,  smoke,  sand,  rain,  or  snow.  In 
addition,  reporting  wind  speed  and  direction  information  at  the  time  images  are 
being  recorded  can  significantly  influence  the  success  of  aviation-related  missions, 
like  those  associated  with  unmanned  aerial  vehicle  take  off,  landing,  and  in-flight 
control. 

2.  Space  and  Time  Scales 

Based  on  an  analysis  by  Meyers, this  section  provides  a  framework  to  help 
categorize  the  spatial  and  temporal  properties  of  recorded  image  data.  Relevant 
time  scales  include,  but  are  not  limited  to,  the  shutter  exposure  time,  time  interval 
between  frames,  time  over  which  images  are  captured  in  a  sequence,  and  time  over 
which  there  is  visual  motion  of  objects  inside  the  field  of  view.  Space  scales 
include,  but  are  not  limited  to,  the  field  of  view,  depth  of  view,  image  resolution, 
pixel  size,  pixel  separation,  scene  color  or  shading  variations  as  a  function  of  spatial 
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location,  spatial  smearing  of  moving  elements  in  the  field  of  view,  spatial  smearing 
due  to  optical  turbulence  and  environmental/weather  effects,  and  smearing  of 
textures  in  the  field  of  view.  Naturally,  the  smearing  of  elements  in  the  field  of  view 
ean  also  be  related  to  the  temporal  resolution  of  the  image  data.  Figure  1  illustrates 
the  primary  space  and  time  scales  that  can  be  used  to  describe  the  various  spatial 
and  temporal  resolutions  of  objects  and/or  activities  in  a  reeorded  image  scene,  to 
include  images  recorded  in  varying  environmental  conditions.  Here,  As  and  At 
represent  changes  in  position  and  time,  respectively. 


Fig.  1  Primary  space  (s)  and  time  (t)  scales 

3.  Image  Resolution,  Image  Context,  and  Identifying  Objects 

To  start,  let  us  explore  the  impact  of  varying  image  resolution  and  image  context 
on  the  analysis  of  an  outdoor  scene.  Try  to  identify  the  3  dome  shapes  in  Fig.  2. 
Without  some  additional  information  related  to  the  object  size,  texture,  or  shape  or 
knowledge  of  how  the  image  context  may  change  in  an  expanded  field  of  view,  it 
is  difficult  to  correctly  identify  and  label  these  familiar  images.  Furthermore, 
distinguishing  various  image  details,  even  in  ideal  conditions  with  regard  to  lighting 
and  visibility,  can  depend  on  the  image  contrast  and  resolution,  where  image 
resolution  here  refers  to  the  number  of  pixels  that  compose  the  image  data  input. 
Interestingly,  Torralba^'  reported  that  for  human  vision  the  brain  can  comprehend 
the  gist  of  an  image  scene  remarkably  quiekly,  regardless  of  whether  low-  or  high- 
resolution  images  are  used.  He  concluded  that  images  at  the  resolution  of  32  x  32 
color  pixels  can  provide  an  observer  enough  information  to  correctly  identify  the 
semantic  category  and  general  layout  of  an  indoor/outdoor  scene.  For  example,  in 
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Fig.  2  the  main  “dome”  eategory  for  these  low-resolution  images  is  identifiable. 
However,  if  we  eonsider  Fig.  3,  whieh  eontains  expanded  fields  of  view  and  higher 
resolution  images  from  whieh  the  elements  in  Fig.  2  were  taken,  then  the  building 
domes  and  many  additional  image  details  ean  be  identified  over  a  mueh  wider  range 
of  spatial  seales. 


What  are  these? 


Fig.  2  Can  you  correctly  identify  these  images?  Image  resolution:  a)  30  x  20  pixels, 
b)  30  X  14  pixels,  and  c)  30  x  16  pixels. 


Fig.  3  Higher-resolution  image  scenes  corresponding  to  the  3  shapes  shown  in  Fig.  2. 
a)  Taj  Mahal  (photo  courtesy  of  desktopdress.com),  b)  US  Capitol  Dome  (photo  courtesy  of 
Library  of  Congress),  and  c)  nuclear  power  plant,  Bushehr,  Iran  (photo  courtesy  of  Behrouz 
Mehri/AFP/Getty  Images). 

To  demonstrate  this  point  further.  Can  you  identify  the  2  similarly  shaped  objects 
shown  in  Fig.  4  without  some  additional  context?  What  if  we  look  at  the  complete 
image  (Fig.  5)  from  which  the  objects  were  taken?  In  this  case,  at  low  resolution,  it 
is  quite  difficult  to  discern  any  individual  elements  in  the  field  of  view.  Of  course, 
the  degree  of  image  resolution  needed  for  a  particular  task  depends  on  the  analysis 
or  computer  vision  problem  of  interest.^’^’'^’^^’^'’^^"^^  Yet,  with  regard  to  scene 
understanding  and  semantic  labeling,  the  slightly  higher  resolution  images  shown 
in  Fig.  6  clearly  provide  more  usable  information.  In  other  words,  when  the  image 
resolution  is  increased  to  64  x  40  pixels  and  greater,  one  can  more  easily  identify 
the  layout  and  main  elements  of  the  image  scene,  such  as  the  reactor  dome  and  hard 
hat  shown  above.  However,  if  still  higher  resolution  images  of  this  reactor  site  are 
considered  (Fig.  7),  then  additional  details  and  information  may  be  gained,  for 
example,  intelligence  relating  to  its  operational  status.  By  analyzing  the  extracted 
and  labeled  objects  shown  in  Fig.  7,  one  might  ask  if  the  reactor  site  is  still  under 
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construction  or  near  completion  as  evidenced  by  the  engineers  wearing  hard  hats, 
the  surveyor,  the  hoist,  and  the  eleetrical  hazard  sign.  Note  here  that  the  hoist, 
surveyor,  and  engineers  wearing  hard  hats  in  the  far-field  of  the  imaged  scene  all 
required  increased  resolution  (i.e.,  >  32  x  32  pixels)  to  be  clearly  identified  (visually 
compare  right  vs.  left  in  Fig.  7).  Table  1  provides  the  various  image  resolution 
details  (in  numbers  of  pixels)  for  the  labeled  objects  in  this  outdoor  scene. 


What  is  it? 

■ — 

A.  Alien  Spaceship 

D.  Temple  Dome 

B.  Stadium  Dome 

E.  Hard  Hat 

C.  Reactor  Dome 

F.  Flying  Saucer 

Fig.  4  Can  you  correctly  identify  these  objects? 


Fig.  5  Low-resolution  images  of  the  scene  from  which  the  objects  in  Fig.  4  were  taken, 
where  neither  large  nor  small  objects  are  discernible  (left:  16  x  10  image  pixels,  right:  32  x  20 
image  pixels) 


Fig.  6  Same  images  as  shown  in  Fig.  5  but  with  a  slightly  higher  resolution  (left:  64  x  40 
image  pixels,  right:  128  x  80  image  pixels) 
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Fig.  7  Same  images  as  shown  in  Figs.  5  and  6  bnt  with  an  even  higher  resolntion  (left:  525 
X  336  image  pixel,  right:  3888  x  2492  image  pixels).  Note  that  the  hoist,  snrveyor,  and  engineers 
wearing  hard  hats  in  the  far-field  of  the  imaged  scene  all  reqnired  increased  image  resolntion 
to  he  clearly  identified. 


Table  1  Image  resolntion  information  (in  nnmbers  of  pixels) 


Object 

Fig.  7  (left) 

Fig.  7  (right) 

Main  image 

525  X  336 

3888  X 2492 

Reactor  dome 

191  x51 

1028  x256 

Hard  hat 

82x45 

405  X  225 

Danger  sign 

32x36 

64x69 

Hoist 

5x24 

39  X  175 

Surveyor 

5x11 

34x74 

Engineers 

5x5 

32x41 

Thus,  we  have  visually  demonstrated  that  image  resolution  and  image  context  play 
an  important  role  in  being  able  to  clearly  recognize  and  identify  individual  objects 
for  scene  understanding.  Here,  object  recognition  may  be  achieved  through 
applications  associated  with  deep  learning  neural  networks. Also,  it  is 
important  to  note  that  image  context  information  can  include  elements  related  to 
time,  geographical  location,  and  environmental  conditions, which  can  help  to 
provide  a  more  detailed  description  of  recorded  scenes  for  indexing  and  future 
retrieval,  as  is  discussed  next. 

4.  Time-  and  Space-Varying  Elements  of  Scene  Understanding 

While  image  resolution  is  a  key  element  for  identifying  objects  and  quickly 
discerning  the  gist  and  general  layout  of  a  recorded  scene,  there  are  many  other 
time-  and  space-varying  elements  that  are  equally  important  for  scene 
understanding  that  should  be  addressed  from  the  very  beginning  of  the  data 
collection  process. 
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4.1  Environmental  Image  Information 


There  are  many  key  pieees  of  information  that  ean  be  identified  as  new  image  data 
are  being  reeorded  that  are  important  and  aeeessible,  but  are  usually  overlooked  or 
left  undoeumented.  For  example,  one  ean  readily  identify  a  timestamp  (relative  to 
the  sun’s  angle  or  relative  to  a  world  eloek),  the  global  positioning  system  (GPS) 
position,  the  prevailing  environmental  and  weather  eonditions,  and  the  field  of 
view,  depth  of  view,  and  image  resolution  (Table  2).  The  first  group,  shown  in 
Table  2,  focuses  on  environmental  information,  such  as  the  GPS  position  and 
altitude  above  ground  level  (AGL),  prevailing  weather,  cloud  cover,  ground  and 
road  conditions,  and  visibility  (e.g.,  fog,  smoke,  haze,  obscurants,  or  optical 
turbulence). 

Identifying  the  key  environmental  and  terrain  conditions  can  provide  location  and 
geographical  context  information  to  help  categorize  image  scenes  recorded  in 
diverse  regions  (e.g.,  coastal,  mountain-valley,  desert,  forest,  urban,  rural,  ocean, 
and  arctic).  Detailed  terrain  characteristics  (e.g.,  muddy,  sandy,  gravelly,  wet,  dry, 
or  icy)  and  reports  of  the  most  current  weather  conditions  available  (e.g.,  rain, 
snow,  fog,  or  haze)  can  be  retrieved  and  annotated  to  help  describe  images  that  may 
be  used  to  support  the  planning  and/or  execution  of  military  operations,  as 
mentioned  above.  Typically,  changing  weather  conditions,  cloud  cover,  and 
visibility  will  bring  about  changes  in  the  illumination  of  a  scene,  which  can  affect 
image  contrast  and  resolution.^^’^'^’^^  Time  of  day  and  sun  angle  information  also 
can  be  retrieved,  which  can  be  useful  to  indicate  when  glare,  shadows,  or  silhouettes 
may  cause  difficulties  for  computer  vision  processes. Also,  taking  note  of 
optical  turbulence  conditions  is  important  because  these  effects  can  significantly 
degrade  and  blur  image  quality  due  to  spatial  smearing. 

The  second  group  in  Table  2  lists  elements  related  to  the  camera  specifications  and 
the  image  data  measurements  themselves  (e.g.,  the  spatial  and  temporal  image 
resolutions,  field  of  view,  depth  of  view,  and  scene  color  or  shading  variations). 
Together  with  the  environmental  information,  these  elements  can  be  used  as  a  basic 
building  block  for  detailed  scene  description  and  image  indexing. 
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Table  2  Time-  and  space-varying  elements  of  scene  nnderstanding 


Environmental  information _ 

GPS  position  and  altitude  AGL 
Location:  geographical  context 
Timestamp 

Weather  conditions,  sky  and  cloud  cover 

Sun/ moon  angle 

Ground/road  conditions 

Visibility 

Vegetation 

Buildings,  parking  lots,  people,  or  crowds _ 

Image/camera  information _ 

Image  resolution 

Pixel  size  and  pixel  separation 

Scene  color  or  shading  variations 

Field  of  view  and  depth  of  view 

Shutter  exposure  time 

Time  interval  between  image  frames 

Time  over  which  images  are  captured  in  a  sequence 


4.2  Scene  Description  Indexing 

Based  on  the  time-  and  space-varying  elements  for  scene  understanding  described 
above,  Table  3  provides  a  top-level  view  for  scene  description  indexing  (i.e.,  these 
are  the  questions  that  one  should  endeavor  to  address  as  image  data  are  being 
recorded  so  that  the  data  can  be  best  indexed  and  retrieved  for  later  use).  In  most 
cases,  the  image  information  (right  column)  can  be  annotated  based  on  the  camera 
type,  lensing,  pixel  array,  and  timing  specifications.  Also,  for  example,  co-located 
range  finder  instrumentation  could  provide  effective  depth  and  field  of  view 
measurements  for  this  purpose.  Additionally,  when  communications  are  available, 
the  most  current  environmental  information  available  (left  column)  can  be  extracted 
from  several  accessible  resources,  such  as  those  shown  in  Table  4.  Obtaining  the 
most  current  data  available  is  advantageous  since  environmental  conditions  (e.g., 
weather  and  terrain)  can  change  over  very  short  temporal  and  spatial  intervals.  For 
example,  access  to  data  from  the  Department  of  Defense  (DOD)  GPS^^  can  provide 
latitude  and  longitude  or  Universal  Transverse  Mercator  (UTM)  location  and 
timestamp  information,  commonly  reported  as  Greenwich  Mean  Time  (GMT)  or 
Coordinated  Universal  Time  (UTC).  Also,  data  from  the  US  Naval  Observatory 
(USNO)^^  can  provide  precise  timing  information  as  well  as  solar  and  lunar 
elevation/azimuth  angles.  Similarly,  terrain  and  geographical  location  and  context 
information  can  be  provided  by  satellite  and  aerial  imagery  from  the  US  Army 
Corps  of  Engineers,  Army  Geospatial  Center  (USACE  AGC)^"^  for  military 
operations  or  from  public  Internet  resources  such  as  Google,^^  MapQuest,^^  Bing,^^ 
and  Yahoo  Maps.^^ 
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Table  3  Scene  description  indexing  (top-level  view) 


Environmental  information 


Image/camera  information 


1  What  is  the  GPS  position  of  the  depicted  scene? 

2  What  is  the  altitude  (AGL)  of  the  depicted  scene? 

3  What  is  the  timestamp  of  the  recorded  image? 

4  What  are  the  current  weather  conditions? 

5  What  is  the  percent  sky/cloud  cover? 

6  What  is  the  sun  elevation/azimuth  angle? 

7  What  is  the  visibility? 

8  What  are  the  current  ground/road  conditions? 

9  What  vegetation  is  in  the  field  of  view? 

10  What  buildings,  parking  lots,  people  or  crowds 
are  in  the  field  of  view? 


What  is  the  camera  field  of  view? 

What  is  the  scene  depth  of  view? 

What  is  the  image  spatial  resolution  (in  pixels)? 

What  is  the  camera  pixel  size? 

What  is  the  scene  color  (R,  B,  G  or  grayscale)? 

What  are  the  spatial  shading  variations  of  the  scene? 

What  is  the  camera  integration  time? 

What  is  the  camera  shutter  exposure  time? 

What  is  the  time  interval  between  frames? 

What  is  the  starting,  ending  and  total  time  for  the  recorded 
image  sequence? 


Table  4  Available/accessible  environmental  image  information 


1  DOD  GPS:  Latitude/longitude  or  UTM,  altitude  (AGL),  GMT,  or  UTC 

2  USNO:  Precise  time,  sun/moon  elevation/azimuth  angle 

3  Terrain  and  location:  USAGE  AGC  —  Satellite/aerial  imagery  and  terrain  analysis 

4  Terrain  and  location:  Google,  MapQuest,  Bing,  Yahoo  Maps 

5  Weather:  USAF  557th  Weather  Wing 

6  Weather:  National  Weather  Service  (NWS)  and  National  Centers  for  Environmental 
Information  (NCEI) 

7  Weather:  Intellicast,  AccuWeather,  Weather  Underground 


Weather  conditions  and  related  oceanic,  atmospheric,  and  geophysical  data  are  also 
available  for  the  military  through  the  US  Air  Force  557th  Weather  Wing^^  (i.e., 
formerly  the  US  Air  Force  Weather  Agency)  and  for  the  civilian  community 
through  the  NWS^°  and  NCEI.^^  Daily  NWS  weather  reports  can  be  found  online 
containing  hourly  records  citing  the  date,  time,  wind  speed  (miles  per  hour), 
visibility  (miles),  weather  (i.e.,  rain,  snow,  fog,  haze,  etc.),  sky/cloud  condition 
(reported  as  overcast  [OVC],  broken  [BRK],  scattered  [SCI],  or  clear  [CLR]  along 
with  the  cloud  ceiling  height  in  hundreds  of  feet  AGL),  air  temperature,  dew  point 
temperature,  relative  humidity  (%),  pressure,  and  precipitation  (in  inches). 
Naturally,  current  weather  and  weather  forecast  information  are  readily  found  on 
Internet  web  sites,  such  as  Intellicast,^^  AccuWeather,’^  and  Weather 
Underground.’"^  Note  however,  that  in  areas  where  communications  are  either 
restricted  or  unavailable,  the  information  needed  to  describe  the  scene,  i.e.,  as 
outlined  in  Table  3,  should  instead  be  gleaned  from  the  recorded  images  when  they 
are  retrieved  for  analysis. 

Thus,  we  have  shown  that  abundant  time-  and  space-varying  environmental  image 
information  can  be  accessed  and  annotated  to  augment  image  data  as  they  are  being 
recorded  for  a  better  organized,  top-down  approach  to  scene  description,  indexing 
and  image  retrieval. 
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5.  Summary  and  Conclusions 


In  this  report,  we  proposed  that  it  is  important  to  incorporate  space-  and  time- 
varying  environmental  image  information  at  the  start  of  the  data  collection  process 
in  order  to  provide  the  end  user  (Soldier)  with  a  better  organized,  top-down 
approach  to  index  and  retrieve  image  data  for  operational  use  and  analysis.  We 
provided  several  examples  to  show  that  space-  and  time-varying  elements  of 
environmental  image  information  (and  changes  in  image  context)  can  be  used  as  a 
basic  building  block  for  detailed  scene  description,  and  as  such,  could  be  used  to 
support  Army  mission  planning  and  execution.  In  conclusion,  we  anticipate  that 
incorporating  space-  and  time-varying  environmental  image  information  in  the  data 
measurement  process  will  lead  to  1)  improved  autonomous  intelligent  systems 
supporting  Army  missions  in  complex  and  changing  environments  and  2)  improved 
course  of  action  strategies  based  on  scene  understanding  (algorithms  and  analysis) 
incorporating  battlefield  environments  changing  in  space  and  time. 
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List  of  Symbols,  Abbreviations,  and  Acronyms 


AGC 

Army  Geospatial  Center 

AGL 

above  ground  level 

BRK 

broken 

CLR 

clear 

DOD 

Department  of  Defense 

GMT 

Greenwich  Mean  Time 

GPS 

global  positioning  system 

NCEI 

National  Centers  for  Environmental  Information 

NWS 

National  Weather  Service 

ovc 

overcast 

SCT 

scattered 

USAGE 

US  Army  Corps  of  Engineers 

USNO 

US  Naval  Observatory 

UTC 

Coordinated  Universal  Time 

UTM 

Universal  Transverse  Mercator 
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